<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head><title></title>
<link href="../style/ebook.css" type="text/css" rel="stylesheet">
</head>
<body>
<h1>Security</h1>
<p>By default, all gates are opened. An easy way to restrict access
to the web application is to do it at the network level, or by using
SSH tunnels.</p>
<p>It is however possible to switch on authentication by either using one of the supplied
backends or creating your own.</p>
<p>Be sure to checkout <a class="reference internal" href="api.html"><span class="doc">Experimental Rest API</span></a> for securing the API.</p>
<div class="section" id="web-authentication">
<h2 class="sigil_not_in_toc">Web Authentication</h2>
<div class="section" id="password">
<h3 class="sigil_not_in_toc">Password</h3>
<p>One of the simplest mechanisms for authentication is requiring users to specify a password before logging in.
Password authentication requires the used of the <code class="docutils literal notranslate"><span class="pre">password</span></code> subpackage in your requirements file. Password hashing
uses bcrypt before storing passwords.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.password_auth
</pre>
</div>
</div>
<p>When password auth is enabled, an initial user credential will need to be created before anyone can login. An initial
user was not created in the migrations for this authentication backend to prevent default Airflow installations from
attack. Creating a new user has to be done via a Python REPL on the same machine Airflow is installed.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># navigate to the airflow installation directory</span>
$ <span class="nb">cd</span> ~/airflow
$ python
Python <span class="m">2</span>.7.9 <span class="o">(</span>default, Feb <span class="m">10</span> <span class="m">2015</span>, <span class="m">03</span>:28:08<span class="o">)</span>
Type <span class="s2">&quot;help&quot;</span>, <span class="s2">&quot;copyright&quot;</span>, <span class="s2">&quot;credits&quot;</span> or <span class="s2">&quot;license&quot;</span> <span class="k">for</span> more information.
&gt;&gt;&gt; import airflow
&gt;&gt;&gt; from airflow import models, settings
&gt;&gt;&gt; from airflow.contrib.auth.backends.password_auth import PasswordUser
&gt;&gt;&gt; <span class="nv">user</span> <span class="o">=</span> PasswordUser<span class="o">(</span>models.User<span class="o">())</span>
&gt;&gt;&gt; user.username <span class="o">=</span> <span class="s1">&apos;new_user_name&apos;</span>
&gt;&gt;&gt; user.email <span class="o">=</span> <span class="s1">&apos;new_user_email@example.com&apos;</span>
&gt;&gt;&gt; user.password <span class="o">=</span> <span class="s1">&apos;set_the_password&apos;</span>
&gt;&gt;&gt; <span class="nv">session</span> <span class="o">=</span> settings.Session<span class="o">()</span>
&gt;&gt;&gt; session.add<span class="o">(</span>user<span class="o">)</span>
&gt;&gt;&gt; session.commit<span class="o">()</span>
&gt;&gt;&gt; session.close<span class="o">()</span>
&gt;&gt;&gt; exit<span class="o">()</span>
</pre>
</div>
</div>
</div>
<div class="section" id="ldap">
<h3 class="sigil_not_in_toc">LDAP</h3>
<p>To turn on LDAP authentication configure your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> as follows. Please note that the example uses
an encrypted connection to the ldap server as you probably do not want passwords be readable on the network level.
It is however possible to configure without encryption if you really want to.</p>
<p>Additionally, if you are using Active Directory, and are not explicitly specifying an OU that your users are in,
you will need to change <code class="docutils literal notranslate"><span class="pre">search_scope</span></code> to &#x201C;SUBTREE&#x201D;.</p>
<p>Valid search_scope options can be found in the <a class="reference external" href="http://ldap3.readthedocs.org/searches.html?highlight=search_scope">ldap3 Documentation</a></p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.ldap_auth

<span class="o">[</span>ldap<span class="o">]</span>
<span class="c1"># set a connection without encryption: uri = ldap://&lt;your.ldap.server&gt;:&lt;port&gt;</span>
<span class="nv">uri</span> <span class="o">=</span> ldaps://&lt;your.ldap.server&gt;:&lt;port&gt;
<span class="nv">user_filter</span> <span class="o">=</span> <span class="nv">objectClass</span><span class="o">=</span>*
<span class="c1"># in case of Active Directory you would use: user_name_attr = sAMAccountName</span>
<span class="nv">user_name_attr</span> <span class="o">=</span> uid
<span class="c1"># group_member_attr should be set accordingly with *_filter</span>
<span class="c1"># eg :</span>
<span class="c1">#     group_member_attr = groupMembership</span>
<span class="c1">#     superuser_filter = groupMembership=CN=airflow-super-users...</span>
<span class="nv">group_member_attr</span> <span class="o">=</span> memberOf
<span class="nv">superuser_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-super-users,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
<span class="nv">data_profiler_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-data-profilers,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
<span class="nv">bind_user</span> <span class="o">=</span> <span class="nv">cn</span><span class="o">=</span>Manager,dc<span class="o">=</span>example,dc<span class="o">=</span>com
<span class="nv">bind_password</span> <span class="o">=</span> insecure
<span class="nv">basedn</span> <span class="o">=</span> <span class="nv">dc</span><span class="o">=</span>example,dc<span class="o">=</span>com
<span class="nv">cacert</span> <span class="o">=</span> /etc/ca/ldap_ca.crt
<span class="c1"># Set search_scope to one of them:  BASE, LEVEL , SUBTREE</span>
<span class="c1"># Set search_scope to SUBTREE if using Active Directory, and not specifying an Organizational Unit</span>
<span class="nv">search_scope</span> <span class="o">=</span> LEVEL
</pre>
</div>
</div>
<p>The superuser_filter and data_profiler_filter are optional. If defined, these configurations allow you to specify LDAP groups that users must belong to in order to have superuser (admin) and data-profiler permissions. If undefined, all users will be superusers and data profilers.</p>
</div>
<div class="section" id="roll-your-own">
<h3 class="sigil_not_in_toc">Roll your own</h3>
<p>Airflow uses <code class="docutils literal notranslate"><span class="pre">flask_login</span></code> and
exposes a set of hooks in the <code class="docutils literal notranslate"><span class="pre">airflow.default_login</span></code> module. You can
alter the content and make it part of the <code class="docutils literal notranslate"><span class="pre">PYTHONPATH</span></code> and configure it as a backend in <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code>.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> mypackage.auth
</pre>
</div>
</div>
</div>
</div>
<div class="section" id="multi-tenancy">
<h2 class="sigil_not_in_toc">Multi-tenancy</h2>
<p>You can filter the list of dags in webserver by owner name when authentication
is turned on by setting <code class="docutils literal notranslate"><span class="pre">webserver:filter_by_owner</span></code> in your config. With this, a user will see
only the dags which it is owner of, unless it is a superuser.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">filter_by_owner</span> <span class="o">=</span> True
</pre>
</div>
</div>
</div>
<div class="section" id="kerberos">
<h2 class="sigil_not_in_toc">Kerberos</h2>
<p>Airflow has initial support for Kerberos. This means that airflow can renew kerberos
tickets for itself and store it in the ticket cache. The hooks and dags can make use of ticket
to authenticate against kerberized services.</p>
<div class="section" id="limitations">
<h3 class="sigil_not_in_toc">Limitations</h3>
<p>Please note that at this time, not all hooks have been adjusted to make use of this functionality.
Also it does not integrate kerberos into the web interface and you will have to rely on network
level security for now to make sure your service remains secure.</p>
<p>Celery integration has not been tried and tested yet. However, if you generate a key tab for every
host and launch a ticket renewer next to every worker it will most likely work.</p>
</div>
<div class="section" id="enabling-kerberos">
<h3 class="sigil_not_in_toc">Enabling kerberos</h3>
<div class="section" id="airflow">
<h4 class="sigil_not_in_toc">Airflow</h4>
<p>To enable kerberos you will need to generate a (service) key tab.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># in the kadmin.local or kadmin shell, create the airflow principal</span>
kadmin:  addprinc -randkey airflow/fully.qualified.domain.name@YOUR-REALM.COM

<span class="c1"># Create the airflow keytab file that will contain the airflow principal</span>
kadmin:  xst -norandkey -k airflow.keytab airflow/fully.qualified.domain.name
</pre>
</div>
</div>
<p>Now store this file in a location where the airflow user can read it (chmod 600). And then add the following to
your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code></p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
<span class="nv">security</span> <span class="o">=</span> kerberos

<span class="o">[</span>kerberos<span class="o">]</span>
<span class="nv">keytab</span> <span class="o">=</span> /etc/airflow/airflow.keytab
<span class="nv">reinit_frequency</span> <span class="o">=</span> <span class="m">3600</span>
<span class="nv">principal</span> <span class="o">=</span> airflow
</pre>
</div>
</div>
<p>Launch the ticket renewer by</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># run ticket renewer</span>
airflow kerberos
</pre>
</div>
</div>
</div>
<div class="section" id="hadoop">
<h4 class="sigil_not_in_toc">Hadoop</h4>
<p>If want to use impersonation this needs to be enabled in <code class="docutils literal notranslate"><span class="pre">core-site.xml</span></code> of your hadoop config.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>&lt;property&gt;
  &lt;name&gt;hadoop.proxyuser.airflow.groups&lt;/name&gt;
  &lt;value&gt;*&lt;/value&gt;
&lt;/property&gt;

&lt;property&gt;
  &lt;name&gt;hadoop.proxyuser.airflow.users&lt;/name&gt;
  &lt;value&gt;*&lt;/value&gt;
&lt;/property&gt;

&lt;property&gt;
  &lt;name&gt;hadoop.proxyuser.airflow.hosts&lt;/name&gt;
  &lt;value&gt;*&lt;/value&gt;
&lt;/property&gt;
</pre>
</div>
</div>
<p>Of course if you need to tighten your security replace the asterisk with something more appropriate.</p>
</div>
</div>
<div class="section" id="using-kerberos-authentication">
<h3 class="sigil_not_in_toc">Using kerberos authentication</h3>
<p>The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to
use it, simply update the connection details with, for example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">&quot;use_beeline&quot;</span>: true, <span class="s2">&quot;principal&quot;</span>: <span class="s2">&quot;hive/_HOST@EXAMPLE.COM&quot;</span><span class="o">}</span>
</pre>
</div>
</div>
<p>Adjust the principal to your settings. The _HOST part will be replaced by the fully qualified domain name of
the server.</p>
<p>You can specify if you would like to use the dag owner as the user for the connection or the user specified in the login
section of the connection. For the login user, specify the following as extra:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">&quot;use_beeline&quot;</span>: true, <span class="s2">&quot;principal&quot;</span>: <span class="s2">&quot;hive/_HOST@EXAMPLE.COM&quot;</span>, <span class="s2">&quot;proxy_user&quot;</span>: <span class="s2">&quot;login&quot;</span><span class="o">}</span>
</pre>
</div>
</div>
<p>For the DAG owner use:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">&quot;use_beeline&quot;</span>: true, <span class="s2">&quot;principal&quot;</span>: <span class="s2">&quot;hive/_HOST@EXAMPLE.COM&quot;</span>, <span class="s2">&quot;proxy_user&quot;</span>: <span class="s2">&quot;owner&quot;</span><span class="o">}</span>
</pre>
</div>
</div>
<p>and in your DAG, when initializing the HiveOperator, specify:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">run_as_owner</span><span class="o">=</span>True
</pre>
</div>
</div>
</div>
</div>
<div class="section" id="oauth-authentication">
<h2 class="sigil_not_in_toc">OAuth Authentication</h2>
<div class="section" id="github-enterprise-ghe-authentication">
<h3 class="sigil_not_in_toc">GitHub Enterprise (GHE) Authentication</h3>
<p>The GitHub Enterprise authentication backend can be used to authenticate users
against an installation of GitHub Enterprise using OAuth2. You can optionally
specify a team whitelist (composed of slug cased team names) to restrict login
to only members of those teams.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.github_enterprise_auth

<span class="o">[</span>github_enterprise<span class="o">]</span>
<span class="nv">host</span> <span class="o">=</span> github.example.com
<span class="nv">client_id</span> <span class="o">=</span> oauth_key_from_github_enterprise
<span class="nv">client_secret</span> <span class="o">=</span> oauth_secret_from_github_enterprise
<span class="nv">oauth_callback_route</span> <span class="o">=</span> /example/ghe_oauth/callback
<span class="nv">allowed_teams</span> <span class="o">=</span> <span class="m">1</span>, <span class="m">345</span>, <span class="m">23</span>
</pre>
</div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If you do not specify a team whitelist, anyone with a valid account on
your GHE installation will be able to login to Airflow.</p>
</div>
<div class="section" id="setting-up-ghe-authentication">
<h4 class="sigil_not_in_toc">Setting up GHE Authentication</h4>
<p>An application must be setup in GHE before you can use the GHE authentication
backend. In order to setup an application:</p>
<ol class="arabic simple">
<li>Navigate to your GHE profile</li>
<li>Select &#x2018;Applications&#x2019; from the left hand nav</li>
<li>Select the &#x2018;Developer Applications&#x2019; tab</li>
<li>Click &#x2018;Register new application&#x2019;</li>
<li>Fill in the required information (the &#x2018;Authorization callback URL&#x2019; must be fully qualified e.g. <a class="reference external" href="http://airflow.example.com/example/ghe_oauth/callback">http://airflow.example.com/example/ghe_oauth/callback</a>)</li>
<li>Click &#x2018;Register application&#x2019;</li>
<li>Copy &#x2018;Client ID&#x2019;, &#x2018;Client Secret&#x2019;, and your callback route to your airflow.cfg according to the above example</li>
</ol>
</div>
<div class="section" id="using-ghe-authentication-with-github-com">
<h4 class="sigil_not_in_toc">Using GHE Authentication with github.com</h4>
<p>It is possible to use GHE authentication with github.com:</p>
<ol class="arabic simple">
<li><a class="reference external" href="https://developer.github.com/apps/building-oauth-apps/creating-an-oauth-app/">Create an Oauth App</a></li>
<li>Copy &#x2018;Client ID&#x2019;, &#x2018;Client Secret&#x2019; to your airflow.cfg according to the above example</li>
<li>Set <code class="docutils literal notranslate"><span class="pre">host</span> <span class="pre">=</span> <span class="pre">github.com</span></code> and <code class="docutils literal notranslate"><span class="pre">oauth_callback_route</span> <span class="pre">=</span> <span class="pre">/oauth/callback</span></code> in airflow.cfg</li>
</ol>
</div>
</div>
<div class="section" id="google-authentication">
<h3 class="sigil_not_in_toc">Google Authentication</h3>
<p>The Google authentication backend can be used to authenticate users
against Google using OAuth2. You must specify the email domains to restrict
login, separated with a comma, to only members of those domains.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.google_auth

<span class="o">[</span>google<span class="o">]</span>
<span class="nv">client_id</span> <span class="o">=</span> google_client_id
<span class="nv">client_secret</span> <span class="o">=</span> google_client_secret
<span class="nv">oauth_callback_route</span> <span class="o">=</span> /oauth2callback
<span class="nv">domain</span> <span class="o">=</span> <span class="s2">&quot;example1.com,example2.com&quot;</span>
</pre>
</div>
</div>
<div class="section" id="setting-up-google-authentication">
<h4 class="sigil_not_in_toc">Setting up Google Authentication</h4>
<p>An application must be setup in the Google API Console before you can use the Google authentication
backend. In order to setup an application:</p>
<ol class="arabic simple">
<li>Navigate to <a class="reference external" href="https://console.developers.google.com/apis/">https://console.developers.google.com/apis/</a></li>
<li>Select &#x2018;Credentials&#x2019; from the left hand nav</li>
<li>Click &#x2018;Create credentials&#x2019; and choose &#x2018;OAuth client ID&#x2019;</li>
<li>Choose &#x2018;Web application&#x2019;</li>
<li>Fill in the required information (the &#x2018;Authorized redirect URIs&#x2019; must be fully qualified e.g. <a class="reference external" href="http://airflow.example.com/oauth2callback">http://airflow.example.com/oauth2callback</a>)</li>
<li>Click &#x2018;Create&#x2019;</li>
<li>Copy &#x2018;Client ID&#x2019;, &#x2018;Client Secret&#x2019;, and your redirect URI to your airflow.cfg according to the above example</li>
</ol>
</div>
</div>
</div>
<div class="section" id="ssl">
<h2 class="sigil_not_in_toc">SSL</h2>
<p>SSL can be enabled by providing a certificate and key. Once enabled, be sure to use
&#x201C;<a class="reference external" href="https://">https://</a>&#x201D; in your browser.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">web_server_ssl_cert</span> <span class="o">=</span> &lt;path to cert&gt;
<span class="nv">web_server_ssl_key</span> <span class="o">=</span> &lt;path to key&gt;
</pre>
</div>
</div>
<p>Enabling SSL will not automatically change the web server port. If you want to use the
standard port 443, you&#x2019;ll need to configure that too. Be aware that super user privileges
(or cap_net_bind_service on Linux) are required to listen on port 443.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Optionally, set the server to listen on the standard SSL port.</span>
<span class="nv">web_server_port</span> <span class="o">=</span> <span class="m">443</span>
<span class="nv">base_url</span> <span class="o">=</span> http://&lt;hostname or IP&gt;:443
</pre>
</div>
</div>
<p>Enable CeleryExecutor with SSL. Ensure you properly generate client and server
certs and keys.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>celery<span class="o">]</span>
<span class="nv">CELERY_SSL_ACTIVE</span> <span class="o">=</span> True
<span class="nv">CELERY_SSL_KEY</span> <span class="o">=</span> &lt;path to key&gt;
<span class="nv">CELERY_SSL_CERT</span> <span class="o">=</span> &lt;path to cert&gt;
<span class="nv">CELERY_SSL_CACERT</span> <span class="o">=</span> &lt;path to cacert&gt;
</pre>
</div>
</div>
</div>
<div class="section" id="impersonation">
<h2 class="sigil_not_in_toc">Impersonation</h2>
<p>Airflow has the ability to impersonate a unix user while running task
instances based on the task&#x2019;s <code class="docutils literal notranslate"><span class="pre">run_as_user</span></code> parameter, which takes a user&#x2019;s name.</p>
<p><strong>NOTE:</strong> For impersonations to work, Airflow must be run with <cite>sudo</cite> as subtasks are run
with <cite>sudo -u</cite> and permissions of files are changed. Furthermore, the unix user needs to
exist on the worker. Here is what a simple sudoers file entry could look like to achieve
this, assuming as airflow is running as the <cite>airflow</cite> user. Note that this means that
the airflow user must be trusted and treated the same way as the root user.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>airflow ALL=(ALL) NOPASSWD: ALL
</pre>
</div>
</div>
<p>Subtasks with impersonation will still log to the same folder, except that the files they
log to will have permissions changed such that only the unix user can write to it.</p>
<div class="section" id="default-impersonation">
<h3 class="sigil_not_in_toc">Default Impersonation</h3>
<p>To prevent tasks that don&#x2019;t use impersonation to be run with <cite>sudo</cite> privileges, you can set the
<code class="docutils literal notranslate"><span class="pre">core:default_impersonation</span></code> config which sets a default user impersonate if <cite>run_as_user</cite> is
not set.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
<span class="nv">default_impersonation</span> <span class="o">=</span> airflow
</pre>
</div>
</div>
</div>
</div>
</body>
</html>